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This study assessed the impact of the Developing Mathematical Ideas (DMI) 
professional development program on grade 4 students’ and teachers’ 
understanding of fractions in eight school districts in three states — Florida, 
Georgia, and South Carolina — during the 2014/15 school year. It found that: 
DMI did not demonstrate any impact on student proficiency in fractions. 
Students of teachers who participated in DMI performed at almost the 
same level as students of teachers who did not participate; the difference 
was not statistically significant. 

The impact of DMI on teachers’ knowledge of fractions was inconclusive. 
Teachers who participated in DMI performed better than teachers who did 
not participate, but the difference was not statistically significant. 
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Summary 


Contemporary state math standards emphasize that students must demonstrate an under- 
standing of the mathematical ideas underlying the computations that have typically been 
the core of the elementary school math curriculum. The standards have put an increased 
emphasis on the study of fractions in upper elementary grades, which are the years during 
which students build a strong foundation in fractions concepts. At the same time, the 
National Mathematics Advisory Panel (2008) cites limited understanding of fractions as 
the key reason for the high failure rate in algebra courses. And longitudinal data from 
both the United States and the United Kingdom have demonstrated that knowledge of 
fractions in the elementary grades plays a powerful role in subsequent success in algebra, 
the gateway to math achievement in high school (Siegler et al., 2012). 

Members of the Regional Educational Laboratory (REL) Southeast Improving Mathemat- 
ics Instruction Research Alliance saw teachers’ lack of in-depth knowledge of fractions 
concepts as a major challenge in their teaching these concepts to their students. Alliance 
members indicated that teachers would benefit from a professional development program 
that focused on building a deep understanding of the mathematical ideas underlying frac- 
tions and of how to apply those ideas in the classroom. This large-scale study investigates 
the effectiveness of such a program to help inform future district and state investments 
in professional development. Members of the REL Southeast Improving Mathematics 
Instruction Research Alliance formed a work group and selected Developing Mathematical 
Ideas (DMI) as the professional development program that seemed best suited to develop 
in-depth teacher knowledge of fractions and that could be scaled up in a large number of 
districts simultaneously. 

Developed by the Education Development Corporation, DMI is designed to help teachers 
think through major mathematical ideas and examine and reflect on how their students 
develop and understand the ideas (Schifter, Bastable, & Russell, 2010a, 2010b). Teachers 
examine vignettes of classroom teaching and examples of student work from their own 
classes and from the classes of other participating teachers. Teachers also work on fractions 
problems designed to promote their own understanding of fractions concepts. 

The primary goal of the study was to assess the impact of DMI on grade 4 students’ and 
teachers’ understanding of fractions. The study was conducted during the 2014/15 school 
year using data from 84 schools in eight school districts in three states (Florida, Georgia, 
and South Carolina). Participants included 4,204 grade 4 students and 264 grade 4 teach- 
ers. Nine trained facilitators provided the professional development. The study used a 
randomized controlled trial, randomly assigning schools to either the treatment condi- 
tion or the control condition. Teachers in schools in the treatment condition received 24 
hours of DMI professional development on fractions during fall 2014. They attended eight 
three-hour sessions conducted over four days (two three-hour sessions per day; one day 
per month). In most cases substitute teachers filled in for teachers during these days; in 
some cases teachers preferred occasional Saturday sessions and were compensated for the 
additional workday. Teachers in the control condition did not receive DMI professional 
development but were free to participate in any type of school- or district-provided profes- 
sional development in math, including fractions. About a third of teachers in the control 
condition indicated that they had participated in some form of professional development 
in fractions. 


The student outcome measure was the Test for Understanding of Fractions (Instructional 
Research Group, 2015), which was administered at the end of the school year to assess 
students’ understanding of fractions concepts and their ability to perform computations 
and word problems. The teacher outcome measure was the Mathematical Knowledge for 
Teaching: Fractions Scale (Learning Mathematics for Teaching, 2008), which was admin- 
istered to all teachers at the end of the study to assess their understanding of the mathe- 
matical ideas involved in teaching fractions and their knowledge of the typical errors and 
misconceptions that can develop as students learn this material. 

Key findings include: 

• DMI did not demonstrate any impact on student proficiency in fractions. Students 
of teachers who participated in DMI performed at almost the same level as stu- 
dents of teachers who did not participate; the difference was not statistically signif- 
icant ( p = .637). 

• The impact of DMI on teachers’ knowledge of fractions was inconclusive. Teach- 
ers who participated in DMI performed 0.19 standard deviation better than teach- 
ers who did not participate, but the difference was not statistically significant 
( P = -051). 

Thus, DMI had nonsignificant impacts on students’ proficiency in fractions and their 
teachers’ knowledge of fractions. The finding of no impact on students’ math proficiency 
is common in the research literature on professional development in math (for example, 
Garet et al., 2011; Garet et al., 2016; Gersten, Taylor, Keys, Rolfhus, & Newman-Gonchar, 
2014). The results suggest that professional development that attempts to build teachers’ 
knowledge of the mathematical ideas underlying the K-8 curriculum, though theoretically 
compelling, does not always lead to improvements in student learning. 
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Why this study? 


In 2013 members of the Regional Educational Laboratory (REL) Southeast Improving 
Mathematics Instruction Research Alliance noted that improving student proficiency in 
fractions was a major priority in the elementary grades. They thought that many teachers 
lacked in-depth knowledge of the often subtle mathematical ideas involved in teaching 
fractions and therefore were unable to go beyond a somewhat superficial emphasis on pro- 
cedures. Alliance members feared that the problem might worsen as states moved to the 
new, more challenging Common Core State Standards in math. 1 

To address this issue, alliance members worked with REL Southeast to design a study to 
determine the impact of Developing Mathematical Ideas (DMI), a professional develop- 
ment program aimed at increasing teachers’ knowledge of fractions, on grade 4 students 
and their teachers. Grade 4 was targeted because grade 4 math emphasizes fractions 
concepts and because the new Common Core State Standards increased the amount of 
instruction time and the complexity of the content to be covered in grade 4. DMI was 
selected because it stresses understanding of the mathematical content and seemed fea- 
sible for large-scale implementation. The study was conducted in three states — Florida, 
Georgia, and South Carolina — during the 2014/15 school year. 

Improving grade 4 students’ proficiency in fractions is a regional priority 

In 2013 only 41 percent of grade 4 students in the United States were deemed proficient 
in math on the National Assessment of Educational Progress. The percentage of students 
who were proficient was below that rate in four of the six states in the REL Southeast 
Region (Alabama, Georgia, Mississippi, and South Carolina) and equal to or slightly 
above in the remaining two states (Florida and North Carolina; National Assessment 
of Educational Progress, 2013). Performance on the latest National Assessment of Edu- 
cational Progress appears similar (National Assessment of Educational Progress, 2015). 
Thus, in most states in the region, grade 4 students have been performing at an inade- 
quate level. 

From an extensive review of the research literature and input from research mathemati- 
cians involved in math education, the National Mathematics Advisory Panel (2008, p. 
xix) concluded that “difficulty with fractions (including decimals and percent) is perva- 
sive and is a major obstacle to further progress in math, including algebra.” The panel 
argued that success in fractions is critical for success in algebra and that a thorough under- 
standing of fractions requires students to confront an array of abstractions that they have 
never experienced before, making fractions, like algebra, “a demonstrable gateway to later 
achievement” (National Mathematics Advisory Panel, 2008, p. xiii). 

Longitudinal data from both the United States and the United Kingdom have demon- 
strated that how well students do in fractions at age 10 (typically grade 4) predicts how 
well they will do in algebra and overall high school math (Siegler et al., 2012). Thus, 
members of the REL Southeast Improving Mathematics Instruction Research Alliance 
viewed success in learning fractions as crucial for increasing students’ success in subse- 
quent algebra courses. 2 
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Building teacher capacity to teach fractions is a regional priority 


Math standards in REL Southeast Region states explicitly say that understanding frac- 
tions and having the ability to articulate the mathematical principles and reasoning that 
underlie fractions computation are as critical as being adept at solving fractions addition, 
subtraction, multiplication, and division problems. Central to understanding fractions are 
concepts such as fractions number sense (a fraction is a number like any other number on 
a number line), fraction magnitude (1/2 is bigger than 1/4), and fractions equivalence (1/3 
is the same as 2/6). Many elementary teachers lack this level of understanding themselves 
(Ball & Forzani, 2009; Ball, Hill, & Bass, 2005). Thus, the REL Southeast Improving 
Mathematics Instruction Research Alliance perceived a need for a professional develop- 
ment program that builds knowledge of the key concepts that underlie operations involv- 
ing fractions. 

Yet Ball (2015) and other researchers stress that skillful math teachers need more than 
content knowledge. They also need specialized knowledge necessary for communicating 
mathematical ideas to their students (Hill et al., 2008). Such understanding enables teach- 
ers to explain and model mathematical ideas and practices, lead math discussions, and 
elicit and interpret students’ rationales for their solutions to a problem (Ball, 2015). These 
teaching practices require teachers to think and analyze student responses on the spot 
(Ma, 1999). 

Selecting a professional development program on fractions to improve students’ and teachers’ 
understanding of fractions 


Skillful math 
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more than content 
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Members of the REL Southeast Improving Mathematics Instruction Research Alliance 
felt that teachers would benefit from a professional development program that focused on 
building a deep understanding of the mathematical ideas underlying fractions and of how 
to apply those ideas in the classroom. Based on the recommendation of its work group 
(which reviewed three potential programs designed to improve teachers’ knowledge of 
math), alliance members selected DMI, a professional development program in fractions 
for grade 4 teachers developed by the Education Development Corporation, because it 
emphasizes learning and understanding of mathematical content and could be delivered 
on a large scale in multiple states. 3 

Evaluating the effectiveness of the selected program could inform future district and 
state investments in professional development programs. Thus, the primary goal of this 
study was to assess the impact of DMI on grade 4 students’ and teachers’ understanding of 
fractions. 


DMI is designed to help teachers think through the major mathematical ideas in elemen- 
tary and middle school and examine how their students develop those ideas (Schifter 
et al, 2010a, 2010b). DMI consists of seven modules focused on various math topics (such 
as whole numbers, fractions, geometry, algebra, and functions). 

The current study focused only on the second DMI module, Making Meaning for Oper- 
ations in the Domains of Whole Numbers and Fractions. This module consists of eight 
three-hour sessions, which are led by a trained DMI facilitator. The majority of the mate- 
rial addresses fractions. Sessions 1 and 2 cover whole number concepts, and sessions 3-8 
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focus on several mathematical themes involving fractions concepts and operations (table 
1). The math content covered in the module is not specific to grade 4. Instead, the module 
is designed to build teacher understanding of a range of whole number and fractions con- 
cepts that are part of the math content taught in the elementary grades. 

Sessions 3-8 emphasize the meaning of various procedures used in fractions arithmetic. 
Participants often solve problems in nontraditional ways and spend time discussing the 
validity of the approaches that their peers develop. 


Table 1. A sampling of themes covered in the Developing Mathematical Ideas module evaluated in the 

study 



Session 

Major mathematical themes 

Grades that sessions 
are aligned with 
based on Common 
Core State Standards 

1: Making meaning for 
whole-number addition 

and subtraction 

"The same situation can be represented by an addition and a subtraction 
sentence.” (p. 15) 

K, 1, 2 

2: Making meaning 
for multiplication and 

division 

“The variety of students' methods for solving story problems involving 
multiplication and division illustrates relationships among operations.” (p. 53) 

2, 3 

3: When dividing doesn’t 
come out evenly 

"The value of a fraction is determined by the relationship between the numerator 
and the denominator.” (p. 87) 

3 


"The same quantity can be represented by different fraction names depending 
on what is taken as 1, the unit or the whole.” (p. 87) 


4: Greater than, less 
than, equal to 

"Multiplying the numerator and the denominator by the same constant yields an 
equivalent fraction.” (p. 117) 

2, 3, 4 


"To determine which of the two fractions is greater, one may find common 
denominators and compare numerators, find common numerators and compare 
denominators, or compare the two fractions to a third number.” (p. 117) 


5: Combining shares, or 
adding fractions 

"Students’ solutions for sharing situations may result in different additive 
expressions with fractions. Teachers can help students develop ideas about 
addition of fractions by challenging them to determine which of their classmates' 
answers are equivalent and which are incorrect.” (p. 149) 

4, 5 


"The equivalence of a -r (b/c) and a x (c/b) can be seen by considering different 
interpretations of a single diagram." (p. 149) 


6: Taking portions of 
portions, or multiplying 

fractions 

"It may be necessary to expand ideas about multiplication of whole numbers in 
order to develop meaning for multiplication involving numbers less than 1.” (p. 
181) 

4, 5, 6 


“Just as multiplication of whole numbers can be represented with a rectangle, so 
can multiplication involving fractions and mixed numbers.” (p. 181) 


7: Expanding ideas about 

division in the context of 

fractions 

"Diagram solutions for problems involving division of fractions can reveal the 
relationships among the operations; that is, a division of fractions problem can 
be solved by calling on addition, subtraction, or multiplication.” (p. 207) 

5, 6 

8: Wrapping up 

"The same basic principles that govern operations with whole numbers are 
called upon to operate with fractions or mixed numbers, but the interpretation of 
each operation may need to be expanded.” (p. 235) 

na 

na is not applicable. 



Note: The Developing Mathematical Ideas module evaluated in the study was Module 2: Making Meaning for Operations in the Domains 
of Whole Numbers and Fractions. Sessions 1 and 2 cover whole number operations. Sessions 3-8 address fractions. 

Source: Authors’ compilation based on Schifter et al. (2010b). 
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In this study most DMI facilitators were elementary math educators who were experienced 
in providing math professional development in their districts. All had been trained by the 
developer and had served previously as facilitators of DMI. 

DMI uses a set of three recurring instructional activities: 

• Sharing student thinking. 

• Case discussions. 

• Math activity. 


During each three-hour session teachers spend 35-45 minutes on the sharing student 
thinking segment, examining and discussing student work samples from their colleagues’ 
classes. During case discussions teachers spend 55-65 minutes reflecting on case studies 
they have read as part of their homework. During math activity teachers spend 35-70 
minutes solving problems that are designed to promote understanding of underlying frac- 
tions concepts (figure 1; see box 1 for further details about the three segments). 

Participating teachers complete reading and writing assignments before each session, 
including collecting and analyzing their students’ work samples on math problems rele- 
vant to the upcoming sessions. Some sessions include additional activities, such as watch- 
ing brief videos of fractions instruction, examining curricular material, and planning for 
student thinking assignments. 


Figure 1. Time is allocated for key instructional activities in each Developing 
Mathematical Ideas session 


Evaluating the 

effectiveness 

of the selected 

professional 

development 

Program- 

Developing 

Mathematical 

Ideas (DMI)-could 

inform future 

district and state 

investments in 

professional 

development 

programs 



Sharing 
student thinking: 

share and discuss 
sample student work 
from their classes 
40 


Note: Total duration of session is 3 hours. On average, Sharing Student Thinking segment is 40 minutes, Case 
Discussions is 60 minutes, and Math Activity is 50 minutes. 

a. Includes such activities as show video, plan student assignment, and plan homework assignment. 

Source: Author’s construction based on Schifter et al. (2010b). 
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Box 1. Key instructional activities in each Developing Mathematical Ideas session 


Sharing student thinking 

In the first segment of each session, teachers examine student work samples shared by other 
teachers in their group and read their colleagues’ analysis of student understanding and mis- 
conceptions. By reading their colleagues’ analyses of student work samples, they deduce 
student understandings and misunderstandings about the underlying mathematical ideas. 
Then, as teachers discuss in their small groups, the facilitator moves from group to group, 
listening and asking questions about student work. In preparation for this activity, all teachers 
assign a problem to their students from Developing Mathematical Ideas (DMI). For example, 
one sample problem given to the students was, “There are 7 brownies to share among 4 
friends. How many brownies would each friend get? Solve this problem with a diagram and 
with an arithmetic sentence. Describe the connections you see between the diagram and the 
arithmetic” (Schifter et al., 2010b, p. 101). 

Case discussions 

During this time, teachers reflect on case studies from the DMI Casebook (Schifter et al., 
2010a), which they have read as part of their homework. The case studies consist of diary 
entries from different grade levels where teachers describe classroom discussions of mathe- 
matical ideas and augment them with pictorial representations of the solutions that students 
used to illustrate their thinking. For example, in one case a teacher describes a math lesson 
that asks students to discuss their thinking as they compare the fractions 1/2 and 2/3 (Schift- 
er et al., 2010a, p. 78). To promote small group discussion, the facilitator provides focus 
questions that draw teachers’ attention to the student work that the case study describes. 
For example, one focus question was, “In the beginning of the case, Harry and Annie use dia- 
grams to explain why 2/3 is greater than 1/2. What ideas about fractions are present in their 
explanations?” (Schifter et al., 2010b, p. 130). As teachers discuss in their small groups, the 
facilitator listens, documents issues, and poses them when the groups convene. 

Math activity 

Teachers solve problems in small groups during this activity. For example, during one session 
teachers must solve the following problem: “Using diagrams, determine which fraction in each 
pair is greater: 4/5 or 4/7; 5/6 or 7/8; 3/8 or 2/9.” To show understanding, teachers solve 
the problems without using standard procedures such as converting the fractions to decimals 
or finding common denominators. The facilitator circulates and interacts with the teachers as 
they complete the problems and attends to those who do not seem to understand. However, 
facilitators do not show participants how to solve a problem. Instead, they ask a series of 
questions that are designed to lead teachers toward understanding the concepts that will help 
them solve the problems successfully by themselves. 


What the study examined 


The study addressed two research questions related to the impact of DMI: 

• What is the impact of teacher participation in DMI on students’ proficiency in 
fractions? 

• What is the impact of teacher participation in DMI on teachers’ knowledge of 
fractions? 
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To answer the questions related to impact, the study team used a randomized controlled 
trial in eight school districts in three states — Florida, Georgia, and South Carolina — 
during the 2014/15 school year. The study sample included 84 schools, 264 teachers, and 
4,204 students. Forty-two schools were assigned to the treatment condition, and 42 schools 
were assigned to the control condition. The 129 teachers from the schools in the treat- 
ment condition received the DMI professional development from September to December 
2014. The 135 teachers from the schools in the control condition did not receive DMI pro- 
fessional development and instead participated in their school’s or district’s typical math 
professional development activities. 

The study team also examined three research questions to help in interpreting the impact 
findings: 

• What was the nature of math professional development attended by teachers who 
did not participate in DMI? 

• How satisfied were teachers who participated in DMI with the professional devel- 
opment they received? 

• Was DMI implemented as intended? 

See box 2 for a summary of the study design, data, and methods and appendix A for further 
details. 


Box 2. Study design, data, and methods 
Study design 

Schools were randomly assigned to treatment and control conditions. For this cluster-randomized 
controlled trial 84 interested schools were matched within each district on five variables (grade 
4 enrollment, percentage of students who exceeded grade 4 math standards, percentage of 
students who were Black, percentage of students who were Hispanic, and percentage of stu- 
dents eligible for the federal school lunch program). One school from each matched pair was 
randomly assigned to the treatment condition (42 schools), and the other was assigned to the 
control condition (42 schools). All grade 4 teachers of math from the randomly assigned schools 
who consented to participate in the study formed the teacher sample (129 from schools in 
the treatment condition and 135 from schools in the control condition). All their students with 
parent consent formed the student sample for the study (2,091 from schools in the treatment 
condition and 2,113 from schools in the control condition). 

Measures 

The study team used total math scaled scores on state math achievement tests (the Florida 
Comprehensive Assessment Test, Georgia’s Criterion-Referenced Competency Tests, and 
South Carolina’s Palmetto Assessment of State Standards) from the spring of grade 3 
to measure students’ entry-level math skills and knowledge. These test scores were used 
because grade 4 students do not know much about fractions at the beginning of the school 
year and might score low on a fractions test, which would hinder the differentiation of students 
needed for a test to be used as a covariate, and because a test of general math could serve 
as a good covariate (since general math skills such as whole number arithmetic predict frac- 
tions performance; Bailey, Siegler, & Geary, 2014; Hansen et al., 2015). The study team used 

(continued) 
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Box 2. Study design, data, and methods (continued) 


the Test for Understanding of Fractions (Instructional Research Group, 2015) to measure the 
student outcome (proficiency in fractions). All students were tested 4-6 weeks before the end 
of the school year, several weeks after the end of the professional development. The test was 
developed by Instructional Research Group, in collaboration with the Institute of Education 
Sciences Center for Improving Learning of Fractions. It includes 26 multiple-choice questions, 
primarily from measures used in the center’s research. Items address foundational fractions 
concepts such as equivalence, magnitude comparison, word problems, and fractions compu- 
tation for addition and subtraction. Two research math educators examined the measure to 
ensure that the wording of the items was clear, the mathematical language was precise and 
accurate, and items covered the full array of grade 4 Common Core State Standards material. 
See appendix B for information on the measure’s reliability. 

The study team used the Mathematical Knowledge for Teaching: Number Concepts and 
Operations scale (Learning Mathematics for Teaching, 2006) to measure entry-level teacher 
knowledge. It includes 28 multiple-choice questions that assess both the content knowledge 
and specialized knowledge needed to teach arithmetic to students in the elementary grades. All 
participating teachers took the pretest prior to the random assignment of schools to the treat- 
ment or control conditions. The study team used the Mathematical Knowledge for Teaching: 
Fractions scale (Learning Mathematics for Teaching, 2008) to measure the teacher outcome 
(knowledge of fractions). It includes 30 multiple-choice questions involving knowledge of frac- 
tions and how to teach fractions to elementary school students. Teachers took the posttest 
4-6 weeks before the end of the school year. See appendix B for information on the reliability 
of the two teacher measures. Sample Mathematical Knowledge for Teaching released items 
are available at http://lmt.mspnet.org/index.cfm/17924. 

Surveys 

Information on the professional development activities of treatment and control teachers was 
gathered using monthly online surveys. The study team gathered information on the professional 
development activities of treatment and control teachers to facilitate a contrast between the 
fractions professional development that the control teachers received and Developing Mathemat- 
ical Ideas (DMI) and any other professional development on fractions that the treatment teachers 
received. The survey items focus on the content and duration of the fractions professional devel- 
opment activities attended by the teachers. See appendix C for a listing of these survey items. 

Teacher satisfaction with DMI was also assessed using online surveys. The study team 
surveyed the treatment teachers to determine teacher satisfaction with DMI at three points in 
time: midway through DMI, at the end of DMI, and at the end of the school year. See appendix 
C for a list of the survey items. 

Teachers and facilitators were surveyed about their education and work experience when 
they consented to participate in the study. The survey included questions about their educa- 
tion, teaching, and mentoring experiences. This information was included in data analysis as 
potential covariates. 

Implementation 

Professional development was provided between September and December, 2014. Participat- 
ing teachers attended the eight three-hour sessions conducted over four days (two three-hour 
sessions per day; one day per month). In most cases substitute teachers filled in for teachers 

(continued) 
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Box 2. Study design, data, and methods (continued) 


during these days; in some cases teachers preferred occasional Saturday sessions and were 
compensated for the additional workday. 

Nine trained DMI staff facilitated all professional development sessions in the study. In 
each district, grade 4 teachers from all schools in the treatment condition met as a group in 
a central location for the DMI sessions. Group sizes ranged from 6 teachers to 33 teachers 
in a group, with larger groups being facilitated by two DMI facilitators. The same facilitators 
facilitated all eight sessions in each district (except one session where last minute scheduling 
conflicts resulted in one of the two facilitators varying across sessions). 

DMI facilitators at each site audio-recorded all eight sessions. The DMI staff developed a 
checklist identifying all key activities for each session, and the study team randomly picked two 
of the eight sessions to check for implementation: session 3 and session 6. Three members 
of the study team listened to the audio-recordings and completed the checklists for sessions 3 
and 6 at each site. To determine how reliable each of the study team members was in complet- 
ing the checklists, a second member assessed completion of the activities for four randomly 
selected sessions (two of session 3 and two of session 6). The average agreement between 
the study team members for these four sessions was 91 percent. 

DMI facilitators recorded teacher attendance for each session. Overall, 80 percent of 
teachers attended all eight DMI sessions and 98 percent of teachers attended six or more 
sessions. The smallest number of sessions attended was three (less than 1 percent of teach- 
ers). Attendance rates varied by district, from 60 percent of teachers attending all eight ses- 
sions in one district to 100 percent attendance in another district (median = 80 percent). 

DMI facilitators also tracked homework completion rates. The average completion rate 
across all assignments was 78 percent (range = 73-81 percent). Completion rates varied by 
district, from 42 percent in one district to 94 percent in another district (median = 83 percent). 

Data analyses 

The study team used multilevel models with analysis of covariance to estimate impacts. Multi- 
level models were used to account for the nested nature of the data (that is, teachers nested 
in schools and students nested in schools). The study team also used analysis of covariance 
to adjust for pretest differences between treatment and control samples. See appendix D for 
more details regarding the analytic models used to determine impacts. 


What the study found 


This study examined the impact of the DMI professional development program on grade 4 
students’ proficiency in fractions and on grade 4 teachers’ knowledge of mathematical con- 
cepts essential for understanding fractions and teaching students about them. Students of 
teachers who participated in DMI performed at almost the same level as students of teach- 
ers who did not participate (impact effect size estimate of -0.03 standard deviation units; 
p = .63). The study also found that the impact of DMI on teachers’ knowledge of fractions 
was inconclusive. Teachers who participated in DMI performed 0.19 standard deviation 
better than teachers who did not participate, but the difference was not statistically signifi- 
cant ( p = .051). Thus, DMI resulted in nonsignificant impacts on both teachers’ knowledge 
of fractions and their students’ proficiency with fractions. 
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Students of teachers who participated in Developing Mathematical Ideas did not demonstrate 
greater knowledge of fractions than did students of teachers who did not participate 


DMI did not have a statistically significant impact on the proficiency in fractions of stu- 
dents of teachers who participated in DMI (table 2). The difference in fractions achieve- 
ment between students of teachers who participated and students of teachers who did not 
participate was miniscule (-0.03 standard deviation). 


Students of teachers who participated in DMI scored slightly higher (16.12) on the posttest, 
Test for Understanding of Fractions, than did students of teachers who did not participate 
(15.84). However, students of teachers who participated in DMI scored slightly higher on 
the pretest (standardized state math tests administered in the spring of grade 3) than did 
students of teachers who did not participate. After this initial difference was statistically 
controlled for, the adjusted posttest scores were similar (16.16 for students of teachers who 
participated in DMI and 16.31 for students of teachers who did not participate), indicating 
that DMI had no impact on student posttest scores (figure 2). 

The effect of DMI on fractions knowledge of teachers who participated in the professional 
development was inconclusive. 

Teachers who participated in DMI scored higher on the posttest, Mathematical Knowledge 
for Teaching: Fractions scale, than did teachers who did not participate (0.19 standard 
deviation units higher; table 3). An effect size of 0.19 corresponds to a difference of 8 per- 
centile points. This means that, on average, knowledge scores of teachers who participated 
in DMI differed from teachers who did not participate by 8 percentile points. However, 
the difference in knowledge of fractions between teachers who participated in DMI and 
teachers who did not participate was not statistically significant ( p = .051). 


DMI did not have 
a statistically 
significant impact 
on the proficiency 
in fractions 
of students of 
teachers who 
participated in 
DMI; its effect 
on fractions 
knowledge of 
teachers who 
participated in 
the professional 
development was 
inconclusive 


Teachers who participated in DMI scored slightly higher on the posttest (20.00) than 
did teachers who did not participate (19.64). However, teachers who participated in DMI 
scored lower on the pretest, Mathematical Knowledge for Teaching: Number Concepts 


Table 2. Pretest and posttest scores of Florida, Georgia, and South Carolina students 
of teachers who participated in the Developing Mathematical Ideas professional 
development program and students of teachers who did not participate, 2014/15 



Pretest 




(standardized 

Posttest 



state math 

(Test for Understanding 



Assessment) 

of Fractions) 



Mean 




score Standard 

Unadjusted Adjusted Standard 

Hedges’ 

Group 

(z-score) deviation 

score score deviation 

g a p value 


Students of teachers who 

participated in DMI (n = 2,091) 

0.05 

1.01 

16.12 

16.16 

5.43 

-0.03 

.637 

Students of teachers who did not 

participate in DMI (n = 2,113) 

-0.05 

0.99 

15.84 

16.31 

5.22 




DMI is Developing Mathematical Ideas. 

a. Hedges’ g is an effect size index that shows the magnitude of the difference between students of teachers 
who participated in DMI and students of teachers who did not participate. 

Source: Authors' analysis of primary data collected for the study; see appendix A. 
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Figure 2. Students of Developing Mathematical Ideas teachers and control teachers 
had similar posttest scores after initial differences were controlled for, 2014/15 


Mean adjusted score on posttest 
25 -i 


20 - 


15 - 

10 - 

5 - 

0 -L 

Students of teachers Students of teachers 

who participated in DMI who did not participate in DMI 




DMI is Developing Mathematical Ideas. 

Note: The posttest is the Test for Understanding of Fractions. 

Source: Authors' analysis of primary data collected for the study; see appendix A. 


Table 3. Pretest and posttest scores of Florida, Georgia, and South Carolina 
teachers who participated in Developing Mathematical Ideas and teachers who did 
not participate, 2014/15 



Pretest 
(Mathematical 
Knowledge for 
Teaching: Number 
Concepts and 
Operations scale) 

Posttest 

(Mathematical Knowledge for 
Teaching: Fractions scale) 



Group 

Mean 

score Standard 
(z-score) deviation 

Unadjusted Adjusted 
score score 

Standard 

deviation 

Hedges’ 

g a 

p value 

Teachers who participated in 

DMI ( n = 129) 

13.60 

4.06 

20.00 20.23 

4.19 

0.19 

.051 

Teachers who did not participate 
in DMI ( n = 135) 

14.10 

4.37 

19.64 19.42 

4.22 




DMI is Developing Mathematical Ideas. 

a. Hedges' g is an effect size index that shows the magnitude of the difference between students of teachers 
who participated in DMI and students of teachers who did not participate. 

Source: Authors' analysis of primary data collected for the study; see appendix A. 
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and Operations, than did teachers who did not participate in DMI. After these initial 
differences were controlled for, adjusted posttest scores were 20.23 for teachers who partici- 
pated in DMI and 19.42 for teachers who did not participate (figure 3); this difference is not 
statistically significant. 


A third of teachers who did not participate in Developing Mathematical Ideas received professional 
development from another source 


Of the 135 teachers who did not participate in DMI, 45 received an average of 5 hours 
of professional development specifically in fractions, compared with 20 hours for teachers 
who participated in DMI. Of these 45 teachers, some reported doing activities similar to 
those in DMI, such as observing videos of math teaching (33 teachers), working with col- 
leagues to plan lessons (35 teachers), discussing student work (25 teachers), and working on 
math problems (35 teachers). 

Most professional development reported by teachers who did not participate in DMI was 
offered during the same time period that DMI was offered (September-December 2014). 
Teachers who did not participate in DMI and some teachers who did participate reported 
receiving other professional development in fractions in the months preceding and fol- 
lowing DMI (August 2014 and January, February, and March 2015). See appendix D for 
additional details on the contrast in professional development received by treatment and 
control teachers. 


Teachers who did 
not participate 
in DMI and some 
teachers who 
did participate 
reported receiving 
other professional 
development in 
fractions in the 
months preceding 
and following DMI 


Figure 3. Teachers who participated in the Developing Mathematical Ideas 
professional development program scored slightly higher on the posttest than did 
teachers who did not participate, 2014/15 

Mean adjusted score on posttest 
25 - 

20 - 

15 - 

10 - 

5 - 

0 -L 

DMI is Developing Mathematical Ideas. 

Note: The posttest is the Mathematical Knowledge for Teaching: Fractions scale. 

Source: Authors' analysis of primary data collected for the study; see appendix A. 



Teachers who Teachers who 

participated in DMI did not participate in DMI 
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Most teachers who participated in Developing Mathematical Ideas were satisfied with the 
professional development they received 

Among teachers who participated in DMI, the mean rating for each of four questions on 
a survey about satisfaction with the program was approximately 3 (agree) on a scale of 1 
(strongly disagree) to 4 (strongly agree). This suggests that teachers saw some benefit to 
participating in DMI. The mean ratings did not change significantly over time, except 
for one item about improvement in their knowledge of how to teach fractions, which did 
improve at a statistically significant level over time (table 4). 

Overall, the majority of teachers who participated in DMI responded “agree” or “strongly 
agree” to all four items. For example, 83 percent of teachers agreed or strongly agreed that 
attending DMI sessions was a good professional learning experience and that their knowh 
edge of fractions had improved. A somewhat smaller percentage of teachers (78 percent) 
agreed or strongly agreed that their knowledge of how to teach fractions had improved. 

Of the eight districts that participated in the study, one district was a strong outlier. In 
contrast to the overall positive attitudes of teachers in other districts, teachers from the 
outlier district were more negative toward DMI (figure 4). Most teachers from this district 
did not see any positive benefits to their participation. These opinions persisted and did 
not change over time. A district administrator indicated that teachers wanted professional 
development that focused more on strategies and techniques that they could take back 
and implement in their classrooms. Despite the dissatisfaction with the overall focus of the 
professional development, attendance was at 100 percent, and homework completion at 
94 percent in this district. 


Among teachers 
who participated 
in DMI, the mean 
rating for each 
of four questions 
on a survey about 
satisfaction with 
the program was 
approximately 3 
(agree) on a scale 
of 1 (strongly 
disagree) to 4 
(strongly agree) 


Table 4. Teacher satisfaction with Developing Mathematical Ideas at three points 
in time, 2014/15 


Mean rating (1, strongly disagree, 
to 4, strongly agree) 

(standard deviation) 

Midway 

through End of F statistic 

professional professional End of the (degrees of 


1 Survey item 

development 

development 

school year 

freedom) 

p value 1 

My knowledge of fractions has 
improved. 

2.94 

(0.69) 

3.12 

(0.74) 

3.05 

(0.69) 

2.15 

(1, 7) 

.186 

My knowledge of how to teach 

2.85 

3.02 

3.02 

11.29* 

.012 

fractions has improved. 

(0.65) 

(0.79) 

(0.74) 

(1, 7) 


Attending the DMI professional 

2.99 

3.09 

3.07 

1.33 

.287 

development sessions was a good 
professional learning experience. 

(0.72) 

(0.79) 

(0.72) 

(1, 7) 


1 was able to put into practice what 

3.01 

3.07 

3.02 

0.02 

.895 

1 learned. 

(0.72) 

(0.73) 

(0.74) 

(1, 7) 



* Significant at p < .05. 

DMI is Developing Mathematical Ideas. 

Note: Of 129 teachers who participated in DMI, 121 responded to all three surveys. 
Source: Authors' analysis of primary data collected for the study; see appendix A. 
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Figure 4. Teachers from one school district were more negative toward the 
Developing Mathematical Ideas professional development program, compared with 
teachers from the other seven districts, 2014/15 


Percent of teachers who responded agree or strongly agree 


m Least satisfied district Mean of remaining six districts ■ Most satisfied district 
(7 teachers total) (110 teachers total) (12 teachers total) 



of fractions of how to 

has improved teach fractions 

has improved 



Attending the I was able to 

DMI professional put into practice 

development sessions what I learned 
was a good professional 
learning experience 


DMI is Developing Mathematical Ideas. 

Note: Values are average satisfaction across surveys at three points in time. 
Source: Authors' analysis of primary data collected for the study. 


Implementation of the Developing Mathematical Ideas professional development varied across the 
eight districts and across sessions 

To examine implementation of session 3 and session 6 at each of the eight districts, the 
study team used a fidelity checklist developed by senior DMI staff. 

The checklist for session 3 included 24 activities necessary for implementing the program 
as designed, and the checklist for session 6 included 17 such activities. According to the 
senior DMI staff, facilitators had to implement at least 80 percent of the activities in each 
session, a requirement that allowed facilitators to implement the session as intended while 
providing room to adjust the session according to participants’ content knowledge and 
understanding of major ideas. 

For session 3, facilitators implemented at least 80 percent of the activities in six of the eight 
districts (the median was 90 percent, and the range was 50-96 percent). In contrast, for 
session 6 facilitators implemented at least 80 percent of the activities in three of the eight 
districts (the median was 76 percent, and the range was 65-100 percent). The two districts 
that had below 80 percent implementation for session 3 also had below 80 percent imple- 
mentation for session 6 (table 5). 

Discussion and follow-up activities toward the end of the sessions tended to be omitted in 
both sessions, though more often in session 6. Because multiplying and dividing fractions 
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Table 5. Percentage of Developing Mathematical Ideas facilitators who completed 
80 percent of activities necessary for implementing the program as designed, by 
district, 2014/15 


1 District 

Session 3 

Session 6 1 

i 

96 

65 

2 

83 

88 

3 

50 

65 

4 

58 

71 

5 

96 

76 

6 

88 

82 

7 

92 

76 

8 

96 

100 

Source: Authors' analysis of primary data collected for the study; see appendix A. 


without relying on standard algorithms are far more difficult than comparing fractions, the 
pace of session 6 may have been slower, not allowing for completion of the full lesson. 

Implications of the study findings 


DMI resulted in nonsignificant impacts on students’ proficiency in and teachers’ under- 
standing of fractions. The lack of findings at the student level fits the pattern emerging 
from other large-scale studies of math professional development, where impacts on student 
achievement have been elusive (Garet et al., 2011; Garet et al., 2016; Gersten et al., 2014). 
The inconclusive impacts on teacher knowledge also seem to fit the pattern emerging from 
this research, where changes at the teacher level have been inconsistent. 


When combined 
with previous 
studies, the 
findings of the 
current study raise 
concerns about the 
effectiveness of 
math professional 
development in 
bringing about 
changes in 
student learning 


When combined with previous studies, the findings of the current study raise concerns 
about the effectiveness of math professional development in bringing about changes in 
student learning. However, several issues arising from this study may provide guidance for 
policy and practice and for future research. 


Time and support to bring about change 

The theory of change in this study was that professional development would bring about a 
change in teacher knowledge, which in turn would affect student learning because teach- 
ers who truly understand the mathematical ideas that underlie the computational proce- 
dures are more likely to be able to explain and clarify possible student misconceptions. 

Teachers may need more time than was available in the study to learn, understand, and 
internalize the fractions content and then transfer it from the professional development 
sessions into the classroom by way of meaningful, high-quality instructional activities. For 
example, knowledge of fractions content may not translate immediately to providing clear 
explanations or identifying areas of student understanding or misunderstanding. School 
district leaders may want to examine the feasibility of providing professional development 
in the summer just prior to the start of the school year to provide more time for partici- 
pants to internalize and apply the knowledge in their classrooms. 
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Teachers may also need additional support beyond participating in the professional devel- 
opment to facilitate the transfer to their classrooms of what they learned during the 
sessions. For example, knowing about the mathematical ideas underlying fractions equiv- 
alence problems may not easily translate to setting up mathematical tasks designed specif- 
ically to highlight those concepts for their students. School district leaders may also want 
to examine the need for additional professional development in the form of coaching or 
guidance through professional learning communities to help teachers translate the knowl- 
edge into effective pedagogical activities (see for example, Gersten et al, 2014). 

Professional development implemented as intended 

The study team assessed two of the audiotaped sessions (sessions 3 and 6) for procedur- 
al fidelity. For session 3, six of the eight districts implemented at least 80 percent of the 
activities, which was considered acceptable. However, for session 6, which covered a more 
difficult topic, fewer than half the districts implemented at least 80 percent of the activ- 
ities. The activities omitted tended to be among the most challenging ones. It is unclear 
whether the low fidelity of implementation in session 6 and to a lesser extent in session 3 
was a result of problems associated with the developers’ ability to scale up, such as ability to 
provide a large number of well-trained facilitators, or a result of other issues encountered by 
facilitators, such as not having sufficient time to complete all the activities. 

During the DMI sessions, teachers explored the content on their own by solving chal- 
lenging problems and reflecting on case studies and student work samples. The underlying 
philosophy, as one facilitator noted, was that “when you figure it out for yourself, you learn 
so much more.” To accomplish this goal, facilitators were required to ask probing questions 
that guided the teachers toward this understanding. They also had to lead discussions so 
that they, along with the teachers, could co-construct the big ideas in each session (for 
example, a fractional part can be named by identifying it as a sum or product of fractions; 
why a x (b/c) produces the same answer as a + ( c/b )). 

However, in listening to the audiotapes for sessions 3 and 6, the study team noted that the 
facilitators did not always pose these questions or have sufficient time available for partici- 
pants to formulate generalizations on the mathematical ideas they were learning. Perhaps, 
as seen from the posttest scores, teachers’ understanding of fractions concepts improved to 
some extent but did not reach a threshold necessary for seriously improving the quality and 
precision in their math instruction. For future implementations, issues relating to scaling 
up and content coverage need to be examined to ensure that there is sufficient time for 
completing key activities. 

Length of professional development sessions 

The developers of DMI asserted that for optimal learning, the preferred option was to conduct 
one session per day. However, most school districts (and frequently the grade 4 teachers at each 
school) consistently indicated a preference for two three-hour sessions per day. The reasons for 
these scheduling arrangements were mainly logistical. Audio recordings of the sessions indi- 
cated that the long workdays dedicated solely to math were fatiguing for teachers and perhaps 
also for facilitators. The fidelity of implementation, as seen, was higher for the morning session 
3 than for the afternoon session 6. District leaders may want to consider whether it is possible 
to implement one session per day as the developers recommended in future implementations. 


Teachers may 
need additional 
support beyond 
participating in 
the professional 
development to 
facilitate the 
transfer to their 
classrooms of 
what they learned 
during the sessions 


15 


Relative merits of the professional development activities 


While the study team did not survey teachers on the activities that were most useful or 
instructive, field notes for informal observations of DMI sessions indicated that teachers 
were most engaged during the math activity segment and least engaged during case dis- 
cussions and sharing student thinking segments. Future research could survey or interview 
teachers on the relative merits of DMI activities and other math professional development 
programs. 


Differentiated professional development 


The wide distribution of scores on the teacher math content knowledge pretest indicates 
that a “one size fits all” professional development program may not have been the most 
suitable approach for the teachers who participated in the study. Informal observations 
of the DMI sessions by the study team revealed that some teachers seemed to truly grasp 
the mathematical ideas, while others seemed to struggle. The findings suggest that dis- 
trict leaders may want to consider differentiated material and instruction for teachers with 
strong math knowledge versus those who have much less initial understanding of the 
content when planning math professional development activities. 

Breadth versus depth 

DMI covered a wide range of fractions content, from foundational aspects such as unit 
fraction and equivalence to multiplication and division of fractions. Thus, the program 
included fractions content from grades 3-6. District leaders may want to consider focusing 
more deeply on fewer concepts, so that teachers develop a deeper understanding of the 
mathematical ideas relevant for teaching their grade 4 students. 


The wide 
distribution of 
scores on the 
teacher math 
content knowledge 
pretest indicates 
that a “one size fits 
all" professional 
development 
program may not 
have been the most 
suitable approach 
for the teachers 
who participated 
in the study 


Limitations of the study 


Although the study sites were both urban and rural, they were not a nationally represen- 
tative sample. Thus, the findings are generalizable only to schools, teachers, and students 
similar to those who participated in the study. 

The results are generalizable to only the DMI professional development program and 
not necessarily to other programs that focus on building teachers’ content knowledge of 
fractions or techniques in using student errors and misconceptions to enhance students’ 
understanding. 

While the self-report data show that some teachers who did not participate in DMI had 
professional development on topics and activities similar to those in DMI, their profes- 
sional development experiences may have been different from those of teachers who par- 
ticipated in DMI. The self-report data also do not provide information on whether the 
professional development received by teachers who did not participate in DMI consisted of 
isolated short duration sessions or was cumulative, building on knowledge from one session 
to the next. 
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Appendix A. Study design and study sample 


This appendix explains the study’s design (matching and randomization) and sample char- 
acteristics (demographic characteristics, baseline equivalence, and attrition). 

Study design 

The research team used a multisite cluster randomized controlled trial and randomly 
assigned schools within districts (Donner & Klar, 2000; Shadish, Cook, & Campbell, 
2002). Schools within each district were first matched and then randomly assigned to 
treatment and control conditions. 

The study team used an optimal greedy matching method (King et al., 2007) to match 
schools because it performs similarly to optimal matching but is less computationally intern 
sive (Bruhn & McKenzie, 2009; Gu & Rosenbaum, 1993). Pairs of schools were formed by 
calculating the Mahalanobis distance between all schools in each district based on five 
baseline observables: grade 4 enrollment, percentage of students who exceeded grade 4 
math standards, percentage of students who were Black, percentage of students who were 
Hispanic, and percentage of students eligible for the federal school lunch program. Then 
pairs were identified sequentially by first choosing the pair with the smallest Mahalanobis 
distance, then choosing the pair with the second smallest, and so on, until the sample was 
completely matched. After the pairs were formed, one school from each pair was randomly 
assigned to the treatment condition. This was done by generating random numbers for 
each school, and then assigning the school with the smaller random number to the treat- 
ment condition. 

Study sample 

The study took place in eight districts across three states (one in Florida, four in Georgia, 
and three in South Carolina). A total of 84 schools agreed to participate in the study (42 
were assigned to the treatment condition and 42 were assigned to the control condition). 
The analytic sample includes 264 teachers and 4,204 students (figure Al). 

No schools dropped out of the study. Overall, 5.4 percent of the teachers left the study. 
Differential attrition between treatment and control teachers was 0.5 percent. Reasons for 
leaving the study across both treatment and control teachers included health and family 
situation, leaving the district, or taking a different teaching position within the district. 
Student attrition could not be determined as the study team was unable to verify which 
students with parent consent were present at many of the schools at the time of random- 
ization. 4 However, in each class all students with parent consent took the posttest, and 
their pretest scores were retrieved from school databases. 

School sample. Of the 84 schools, 14 were rural schools, and 70 were urban schools (table 
Al). Treatment and control schools did not differ significantly on the percentage of grade 
4 students who passed the state math test or on any of the demographic characteristics 
(table A2). 

Teacher sample. Most teachers were female, had more than 11 years of teaching experi- 
ence, and had 2 to 5 years of experience teaching grade 4 students (table A3). The number 
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Figure Al. Formation of teacher and student samples 



Source: Authors' analysis of primary data collected for the study. 


Table Al. Geographic location of schools in the study (number of schools) 


Location of 
school district 

Florida 

Georgia 

South Carolina 

Treatment 

group 

Control 

group 

Treatment 

group 

Control 

group 

Treatment 

group 

Control 

group 

Rural 

na 

na 

2 

3 

5 

4 

Urban 

City 3 

na 

na 

9 

10 

9 

9 

Suburb 6 

6 

6 

5 

3 

6 

6 

Town, fringe 6 

na 

na 

na 

1 

na 

na 


na is not applicable. 

a. Territory inside an urbanized area and inside a principal city. 

b. Territory outside a principal city and inside an urbanized area. 

c. Territory inside an urban cluster that is less than or equal to 10 miles from an urbanized area. 

Source: Authors' analysis based on 2014-2015 data from U.S. Department of Education (n.d. a, n.d. b). 
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Table A2. Baseline equivalence of the 84 schools in the study, 2014/15 



Mean 

(standard deviation) 



Characteristic 

Treatment 
condition 
(n = 42) 

Control 
condition 
(n = 42) 

Hedges' g a 

p value 

Number of students in grade 4 

97.83 

98.69 

-0.02 

.755 


(32.05) 

(40.80) 



Percentage of grade 4 students who 

74.37 

74.60 

-0.02 

.925 

passed the state math test 

(11.21) 

(10.55) 



Percentage of students who were Black 

36.86 

36.21 

0.02 

.918 


(28.80) 

(28.57) 



Percentage of students who were Hispanic 

12.41 

15.85 

-0.21 

.332 


(15.89) 

(16.45) 



Percentage of students eligible for the 

61.72 

66.84 

-0.20 

.354 

federal school lunch program 

(26.97) 

(23.27) 



Percentage of students who were English 

8.10 

9.95 

-0.13 

.558 

learner students 

(14.74) 

(14.14) 




Hedges' g is an effect size index that shows the magnitude of the difference between treatment group and 
control group. 

Source: Authors' analysis based on data from the Florida Department of Education, the Georgia Department 
of Education, and the South Carolina Department of Education. 


of teachers with a bachelor’s degree or master’s degree was comparable for the two samples. 
More treatment teachers than control teachers had worked as math coaches, and this dif- 
ference was statistically significant. 

Teachers in the treatment condition scored 0.12 standard deviation below teachers in the 
control condition on the math pretest (Mathematical Knowledge for Teaching: Number 
Concepts and Operations). However, the difference was not statistically significant (table 
A4). The groups showed other differences (such as the number of years of teaching grade 
4 and the number of courses taken in a teaching math type of certification) that were also 
nonsignificant. 

Teachers in the study used a variety of math curricula (table A5). Curriculum variations 
are by district, and therefore the curriculum is the same for teachers in the treatment con- 
dition and teachers in the control condition. 

Student sample. Student state math achievement scores from the spring of grade 3 and 
student demographic data were collected from district databases. Treatment students 
scored 0.10 standard deviation higher on the math pretest than did control students, but 
the difference was not significant (table A6). However, students in the treatment and 
control conditions differed significantly on the percentage of Hispanic students and the 
percentage of talented and gifted students. 

Facilitator sample. Nine facilitators, provided by the developers of Developing Mathe- 
matical Ideas (DMI), facilitated the DMI sessions in the study. All were experienced DMI 
facilitators, with an average 11 years of experience (table A7). Each facilitator was respon- 
sible for facilitating all DMI sessions at each school district site. Five of the nine facilitators 
facilitated sessions in one school district each. Each of the remaining four facilitators facil- 
itated sessions at two school district sites. 
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Table A3. Demographic characteristics of the analytic sample of 264 teachers, 2014/15 



Percent 


Chi-square 


Characteristic 

Total sample 
(N = 264) 

Treatment 
condition 
(n = 129) 

Control 
condition 
(n = 135) 

test 

(degrees of 
freedom) 

p value 

Gender 

Female 

93.94 

94.57 

93.33 

0.178 (1) 

.673 

Degree type 

Bachelor’s 

47.35 

46.51 

48.15 

0.111 (2) 

.946 

Master's 

44.70 

45.74 

43.70 



Coursework beyond master’s 

7.95 

7.75 

8.15 



Total years of classroom teaching 

0-1 

11.74 

10.85 

12.59 

2.102 (3) 

.552 

2-5 

13.64 

11.63 

15.56 



6-10 

22.35 

25.58 

19.26 



11 or more 

52.27 

51.94 

52.59 



Total years of teaching grade 4 

0-1 

30.30 

27.13 

33.33 

1.608 (3) 

.658 

2-5 

40.15 

43.41 

37.04 



6-10 

19.70 

20.16 

19.26 



11 or more 

9.85 

9.30 

10.37 



Has experience teaching math in grades 6-12 a 

Yes 

5.70 

5.43 

5.97 

0.036 (1) 

.850 

No 

94.30 

94.57 

94.03 



Certifications 11 

Elementary 

98.11 

97.67 

98.52 

0.980 (3) 

.806 

Math 

6.82 

6.20 

7.41 



Middle school 

16.67 

14.73 

18.52 



Special education 

6.06 

6.98 

5.19 



Number of undergraduate or graduate classes in math c 

0 

6.11 

6.20 

6.02 

0.141 (3) 

.987 

1-2 

41.98 

42.64 

41.35 



3-5 

42.75 

42.64 

42.86 



6 or more 

9.16 

8.53 

9.77 



Number of undergraduate or graduate classes in methods for teaching math d 

0 

11.49 

10.08 

12.88 

2.129 (3) 

.546 

1-2 

62.45 

61.24 

63.64 



3-5 

21.46 

24.81 

18.18 



6 or more 

4.60 

3.88 

5.30 



Worked as a math coach or mentor in the past three years e 

Yes 

8.08 

11.72 

4.55 

4.504* (1) 

.034 

No 

91.92 

88.28 

95.45 



* Significant at p < .05. 






Note: Percentages may not sum to 100 because of rounding. 




a. Data for one teacher in the control condition were missing. 




b. Percentages do not sum to 100 because some teachers have multiple certifications. 


c. Data for two teachers in the control condition were missing. 




d. Data for three teachers in the control condition were missing. 




e. Data for one teacher in the treatment condition and three teachers in the control condition were missing. 

Source: Authors' analysis of primary data collected for the study. 
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Table A4. Baseline equivalence on math pretest and demographic characteristics of the 264 teachers 
in the analytic sample, 2014/15 


Characteristic 

Mean 

(standard deviation) 

Treatment Control 

condition condition 

(n = 129) (n = 135) 

Hedges’ g 

Cox's d 

Chi-square 

test 

p value 

Score on math pretest — Mathematical 
Knowledge for Teaching: Number Concepts 
and Operations 

13.60 

(4.06) 

14.10 

(4.37) 

-0.12 

na 

na 

.352 

Years of teaching grade 4 

4.14 

(3.28) 

3.85 

(3.35) 

0.09 

na 

na 

.582 

Number of courses taken in math 

1.53 

(0.74) 

1.54 

(0.77) 

-0.01 

na 

na 

.865 

Number of courses taken in teaching math 

1.22 

(0.68) 

1.13 

(0.72) 

0.13 

na 

na 

.290 

Percentage with a master's degree 3 

53.49 

(50.07) 

51.85 

(50.15) 

na 

0.04 

0.07 

.790 

Percentage who taught math in grades 6-12 a 

5.43 

(22.74) 

5.93 

(23.70) 

na 

-0.06 

0.03 

.861 

Percentage with math certification 3 

6.20 

(24.21) 

7.41 

(26.29) 

na 

-0.12 

0.15 

.698 

Percentage with middle school certification 3 

14.73 

(35.58) 

18.58 

(38.99) 

na 

-0.17 

0.68 

.409 

Percentage with special education 

certification 3 

6.98 

(25.57) 

5.19 

(22.26) 

na 

0.19 

0.37 

.542 


na is not applicable. 

Note: Both Hedges’ g and Cox’s d are effect size indexes that show the magnitude of the difference between treatment group and con- 
trol group. Hedges' g is used for continuous measures, and Cox's d is used for dichotomous measures. 

a. Dichotomous measures, for which both Cox's d and chi-square tests are reported. 

Source: Authors' analysis of primary data collected for the study. 


Table A5. Math curriculum use by district, 2014/15 

1 District 

Math curriculum used I 

i 

Envision 

2 

Go Math 

3 

Singapore Math 

4 

Math Expressions 

5 

Envision/V-Math 

6 

Investigations 

7 

Go Math 

8 

My Math 

Source: Authors' analysis of primary data collected for the study. 






Table A6. Baseline equivalence of the 4,204 students in the analytic sample, 2014/15 (percent 


unless otherwise indicated) 


Mean 

(standard deviation) 





Characteristic 

Treatment 
condition 
(n = 2,091) 

Control 
condition 
(n = 2,113) 

Hedges’ g 

Cox's d 

Chi-square 

test 

p value 

Math pretest (z-score) 

0.05 

(1.01) 

-0.05 

(0.99) 

0.09 

na 

na 

.334 

Male 3 

51.32 

(49.99) 

50.83 

(50.00) 

na 

0.01 

0.10 

.752 

Black 3 

31.88 

(46.61) 

29.15 

(45.46) 

na 

0.08 

3.69 

.055 

Hispanic 3 

13.21 

(33.87) 

16.23 

(36.88) 

na 

-0.15 

7.36** 

.006 

White 3 

48.36 

(49.98) 

48.75 

(50.00) 

na 

-0.01 

0.07 

.797 

In special education 3 

8.94 

(28.54) 

8.85 

(28.41) 

na 

0.01 

0.01 

.916 

English learner students 3 

10.23 

(30.32) 

11.97 

(32.47) 

na 

-0.11 

3.22 

.073 

Talented and gifted 3 

8.56 

(27.98) 

11.36 

(31.47) 

na 

-0.19 

9.17** 

.003 


** Significant at p < .01. 
na is not applicable. 

Note: Both Hedges’ g and Cox’s d are effect size indexes that show the magnitude of the difference between the treatment group and 
control group. Hedges’ g is used for continuous measures, and Cox's d is used for dichotomous measures. 

a. Dichotomous measures, for which both Cox's d and chi-square tests are reported. 

Source: Authors' analysis of primary data collected for the study. 
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Table A7. Demographic characteristics of Developing Mathematical Ideas 
facilitators, 2014/15 

Characteristic 

Mean 

Median 

Range 

Number of 
facilitators 
(N = 9) 

Total years 

Total years in position 

11.6 

12 

5-22 

na 

Total years of classroom teaching 

26.2 

25 

7-40 

na 

Total years of teaching math grades 6-12 

5.6 

1 

0-40 

na 

Total years of teaching grade 4 

5.9 

5 

0-25 

na 

Total years of providing math in-service for districts 

16.4 

16 

10-25 

na 

Total years as DMI facilitator 

11.3 

15 

4-16 

na 

Certifications 

Elementary 

na 

na 

na 

8 

Math 

na 

na 

na 

4 

Middle school 

na 

na 

na 

4 

Administration 

na 

na 

na 

2 

National board certified 

na 

na 

na 

3 

Highest degree earned 

Bachelor’s 

na 

na 

na 

1 

Master's 

na 

na 

na 

3 

Coursework beyond master’s 

na 

na 

na 

5 

Current position 

Classroom teacher 

na 

na 

na 

1 

Math coach, specialist, or instructional support 

na 

na 

na 

6 

Consultant 

na 

na 

na 

2 

DMI is Developing Mathematical Ideas; na is not applicable. 




Source: Authors' analysis of primary data collected for the study. 
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Appendix B. Reliability of measures 


The study team examined the internal consistency reliability of the measures used in the 
study using two types of analyses: 

• Classical test score analysis to examine item to total correlations and compute 
coefficient alphas for both measures. 

• Item response theory analysis using a two-parameter logistic model to examine 
items along two parameters: discrimination and difficulty. 

Internal consistency (coefficient alpha) reliability was .86 for the student measure, .72 for 
the teacher pretest measure, and .73 for the teacher posttest measures. All are well over the 
threshold for an outcome measure established by What Works Clearinghouse (2014). 

Test for Understanding of Fractions (student posttest) 

Based on data from a sample of 5,005 grade 4 students, Cronbach’s alpha was .86 for 
the 26-item Test for Understanding of Fractions, which was used as the student posttest 
measure (table Bl). Item response theory analysis indicated that reliability was greater than 
.80 from an ability level of -2.4 to 1.6 and greater than .90 from an ability level of -0.8 to 
0.4. Maximum information for the 26 items (that is, the peak of the information curve) 
was at an ability level of 0 (I[0] = 10.62). 

Item-level percentages for a correct response ranged from 30 percent for item 23 to 
95 percent for item 2 (table B2). Of the 26 items, 24 had discrimination estimates above 
0.60, ranging from 0.75 to 2.13. Items with higher discrimination estimates (that is, steeper 
slopes) are considered more informative. Item difficulty estimates ranged from -9.94 for 
item 3 to 0.73 for item 23. Items with negative difficulty estimates are considered easier 
than items with positive difficulty estimates. 

Mathematical Knowledge for Teaching: Number Concepts and Operations scale (teacher pretest) 

Based on a sample of 279 grade 4 teachers, the Cronbach’s alpha for the 28-item Mathe- 
matical Knowledge for Teaching: Number Concepts and Operations scale, which was used 
as the teacher pretest measure, was .72 (table B3). Based on item response theory analysis, 
the reliability was greater than .75 from an ability level of roughly -2.0 to 2.4 and greater 
than .80 from an ability level of -0.4 to 1.2. Maximum information (that is, the peak of 
the information curve) for the 28 items was at an ability level of 0.4 (I[0] = 4.43). 


Table Bl. Reliability of the student posttest — Test for Understanding of Fractions, 


2014/15 

Measure 

Cronbach's alpha from 
classical test theory analysis 

Reliability estimates from item 
response theory analysis 

Test for Understanding of Fractions 

.86 

> .80 


Note: Analysis based on a sample of 5,005 students. Item response theory reliabilities were determined using 
a two-parameter logistical model based on the following parameters: discrimination and difficulty. 

Source: Authors' analysis of primary data collected for the study; see appendix A. 
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Table B2. Item-level analysis of the student posttest — Test for Understanding of 


Fractions, 

, 2014/15 







Cronbach’s alpha 

Item response theory analysis 

Item 

number 

Percent correct 

Item-total 

correlation 

Alpha if item 
deleted 

Discrimination 

Difficulty 

1 

44.10 

-.24 

.86 

-0.56 

-0.46 

2 

95.06 

.18 

.85 

1.02 

-3.30 

3 

89.33 

.06 

.85 

0.22 

-9.94 

4 

71.45 

.39 

.84 

1.20 

-0.99 

5 

47.43 

.52 

.83 

1.57 

0.06 

6 

49.71 

.29 

.84 

0.75 

0.01 

7 

42.14 

.47 

.84 

1.35 

0.29 

8 

48.93 

.48 

.84 

1.43 

0.01 

9 

76.20 

.33 

.84 

0.97 

-1.43 

10 

34.44 

.57 

.83 

2.06 

0.49 

11 

73.65 

.30 

.84 

0.87 

-1.38 

12 

87.57 

.35 

.84 

1.69 

-1.64 

13 

57.70 

.52 

.83 

1.75 

-0.31 

14 

52.70 

.48 

.84 

1.38 

-0.14 

15 

87.53 

.25 

.84 

0.93 

-2.40 

16 

64.04 

.30 

.84 

0.79 

-0.84 

17 

85.17 

.30 

.84 

1.14 

-1.88 

18 

61.10 

.51 

.83 

1.68 

-0.44 

19 

77.74 

.29 

.84 

0.81 

-1.74 

20 

47.49 

.41 

.84 

1.11 

0.09 

21 

39.98 

.52 

.83 

1.60 

0.34 

22 

55.18 

.54 

.83 

1.91 

-0.22 

23 

30.33 

.50 

.84 

1.61 

0.73 

24 

70.51 

.38 

.84 

1.08 

-1.01 

25 

37.92 

.59 

.83 

2.13 

0.36 

26 

54.21 

.52 

.83 

1.56 

-0.19 


Source: Authors' analysis of primary data collected for the study; see appendix A. 


Table B3. Reliability of the teacher pretest — Mathematical Knowledge for 
Teaching: Number Concepts and Operations scale, 2014/15 


Measure 

Cronbach’s alpha from 
classical test theory analysis 

Reliability estimates from Item 
Response Theory analysis 

Mathematical Knowledge for Teaching: 
Number Concepts and Operations scale 

.72 

> .75 


Note: Psychometric data based on a sample of 279 teachers. Item response theory reliabilities were deter- 
mined using a two-parameter logistical model based on the following parameters: discrimination and difficulty. 

Source: Authors' analysis of primary data collected for the study; see appendix A. 


Item discrimination parameters for the 28 items ranged from 0.24 (item 15) to 1.40 (item 
16). Items with higher discrimination estimates (that is, steeper slopes) are considered more 
informative. Item difficulty estimates range from -3.09 (item 1) to 5.53 (item 27). Items 
with negative difficulty estimates are considered easier than items with positive item diffr- 
culty estimates (table B4). 
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Table B4. 

Item-level analysis of the teacher pretest 

— Mathematical Knowledge for 

Teaching: 

Number Concepts and Operations scale, 

2014/15 


Item 


Cronbach’s alpha 

Item response theory analysis 

number 

Item total correlation Alpha if item deleted Discrimination 

Difficulty 

1 

.12 

.72 

0.31 

-3.09 

2 

.26 

.71 

0.72 

-1.65 

3 

.20 

.71 

0.44 

-0.29 

4 

.19 

.71 

0.56 

-2.92 

5 

.29 

.71 

1.06 

1.78 

6 

.15 

.72 

0.45 

1.52 

7 

.36 

.70 

1.16 

0.20 

8 

.27 

.71 

0.97 

1.76 

9 

.20 

.71 

0.67 

-2.01 

10 

.34 

.70 

1.04 

0.49 

11 

.33 

.70 

1.13 

-0.27 

12 

.22 

.71 

0.39 

-0.56 

13 

.22 

.71 

0.91 

2.18 

14 

.31 

.70 

0.96 

0.25 

15 

.18 

.71 

0.24 

-1.60 

16 

.42 

.70 

1.40 

0.41 

17 

.30 

.71 

0.96 

-0.86 

18 

.25 

.71 

0.86 

-1.44 

19 

.21 

.71 

0.53 

0.80 

20 

.26 

.71 

0.73 

0.68 

21 

.15 

.72 

0.44 

3.34 

22 

.13 

.72 

0.50 

-3.11 

23 

.33 

.70 

1.06 

-1.62 

24 

.23 

.71 

0.58 

0.36 

25 

.21 

.71 

0.53 

0.83 

26 

.21 

.71 

0.71 

-2.78 

27 

.11 

.72 

0.41 

5.53 

28 

.32 

.71 

1.14 

1.73 


Source: Authors’ analysis of primary data collected for the study; see appendix A. 


Mathematical Knowledge for Teaching: Fractions scale (teacher posttest) 

Based on a sample of 266 grade 4 teachers, the Cronbach’s alpha for the 30'item Mathe- 
matical Knowledge for Teaching: Fractions scale, which was used as the teacher posttest 
measure, was .73 (table B5). Item response theory analysis indicated that reliability was 
greater than .80 from an ability level of -2.8 to an ability level of 0.4. Maximum informa- 
tion for the 30 items (that is, the peak of the information curve) was at an ability level of 
-2.0 (I[0] = 6.05). 

Item discrimination parameters for the 30 items ranged from 0.06 (item 28) to 1.99 (item 
10). Items with higher discrimination estimates (that is, steeper slopes) are considered 
more informative. Item difficulty estimates for the data ranged from -31.18 (item 28) to 
3.63 (item 24). Items with negative difficulty estimates are considered easier than items 
with positive difficulty estimates (table B6). 
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Table B5. Reliability of teacher posttest — Mathematical Knowledge for Teaching: 

Fractions scale, 2014/15 



Measure 

Internal consistency of items 
Cronbach’s alpha from 
classical test theory analysis 

Reliability estimates from item 
response theory analysis 

Mathematical Knowledge for Teaching: 

Fractions scale 

.73 

> .80 


Note: Psychometric data based on a sample of 266 teachers. Item response theory reliabilities were deter- 
mined using a two-parameter logistical model based on the following parameters: discrimination and difficulty. 

Source: Authors' analysis of primary data collected for the study; see appendix A. 


Table B6. Item-level analysis of teacher posttest — Mathematical Knowledge for 
Teaching: Fractions scale, 2014/15 


Item 


Cronbach’s alpha 

Item Response Theory analysis 

number 

Total item correlation Alpha if item deleted 

Discrimination 

Difficulty 

1 

.23 

.72 

0.62 

-1.01 

2 

.32 

.72 

0.98 

-1.45 

3 

.25 

.72 

0.67 

-0.87 

4 

.31 

.72 

0.89 

0.89 

5 

.22 

.72 

0.50 

0.22 

6 

.17 

.73 

0.39 

0.59 

7 

.19 

.73 

1.66 

-2.66 

8 

.25 

.72 

1.34 

-2.27 

9 

.20 

.73 

1.28 

-2.58 

10 

.21 

.73 

1.99 

-2.57 

11 

.38 

.71 

1.30 

-0.55 

12 

.36 

.71 

1.37 

-1.19 

13 

.05 

.74 

0.18 

-7.64 

14 

.16 

.73 

0.47 

-2.76 

15 

.38 

.71 

1.38 

-0.74 

16 

.17 

.73 

0.49 

1.72 

17 

.38 

.71 

1.33 

-0.87 

18 

.11 

.73 

0.41 

-4.91 

19 

.20 

.72 

1.08 

2.71 

20 

.28 

.72 

1.08 

-1.47 

21 

.33 

.72 

1.06 

-1.42 

22 

.25 

.72 

0.62 

-1.49 

23 

.43 

.71 

1.28 

0.33 

24 

.18 

.73 

0.56 

3.63 

25 

.15 

.73 

0.40 

-1.86 

26 

.24 

.72 

0.76 

-2.13 

27 

.32 

.72 

0.83 

-0.11 

28 

.02 

.73 

0.06 

-31.18 

29 

.29 

.72 

0.81 

-1.32 

30 

.29 

.72 

0.85 

-1.30 


Source: Authors’ analysis of primary data collected for the study; see appendix A. 
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Appendix C. Online surveys 


This appendix presents the two online surveys used in this project. One survey gathered 
information on the professional development activities of teachers who participated in 
Developing Mathematical Ideas and of teachers who did not participate. The other survey 
gauged satisfaction with the program among teachers who participated in DMI. 

Survey of professional development activities 

To facilitate a contrast between the professional development activities of teachers who 
participated in DMI and of teachers who did not participate, the study team surveyed all 
teachers in the study to gather information about their professional development activities 
in math (table Cl). The team administered the survey online each month, from Septem- 
ber 2014 to April 2015. Teachers received an email reminder each month to complete the 
survey. On average, the response rate for the eight surveys was 91 percent. The survey 
items were drawn from previous professional development research that the study team 
conducted (Gersten, Dimino, Jayanthi, Kim, & Santoro, 2010) and from the teacher survey 
used in the Middle School Mathematics Professional Development Study by Garet et al. 
( 2011 ). 

Survey of teacher satisfaction with Developing Mathematical Ideas 

To gauge treatment teacher satisfaction with DMI, the study team surveyed the teachers 
three times: midway through the professional development, at the end of the professional 
development, and at the end of the school year. The average response rate for the three 
surveys was 97 percent. The study team developed the survey items and administered them 
to the teachers online (table C2). 


Table Cl. Monthly fractions professional development survey questions given to treatment and control 
teachers 

1 Fractions professional development questions 

Response B 

1. Did you participate in math professional development? 

□ Yes □ No 

2. Was the topic of fractions covered in the math professional development you participated in? 

□ Yes □ No 

3. For how many hours did the professional development you attended cover fractions? 

hours 

4. Which of the following topics were included in the fractions portion of the professional development? Check all that apply. 

Understanding fractions as numbers 

□ 

Equivalence of fractions 

□ 

Fraction addition/subtraction 

□ 

Fraction multiplication/division 

□ 

Improper fractions/mixed numbers 

□ 

5. Check the activities that were part of the math professional development sessions you attended. Check all that apply. 

Observing videos or live demonstrations of mathematics teaching during the professional development session 

□ 

Working with colleagues to plan or enhance mathematics lessons/activities 

□ 

Discussion of students' mathematics work samples with colleagues to analyze how students think about and 
learn mathematics (including common student difficulties and misconceptions) 

□ 

Working on mathematics problems (either individually or with a partner) 

□ 

Engaging in group discussions about the mathematics concepts we were learning 

□ 


(continued) 
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Table Cl. Monthly fractions professional development survey questions given to treatment and control 
teachers (continued) 

1 Fractions professional development questions 

Response 1 

6. Check the follow-up activities that were part of the math professional development you attended. Check all activities that apply. 

1 was required to observe demonstrations of mathematics teaching in classroom settings 

□ 

1 was required to develop mathematics lessons/activities outside of the professional development session, 
for use in my classroom 

□ 

1 was required to implement a mathematics lesson or activity in my classroom 

□ 

1 received feedback from the professional development staff about my understanding of mathematical ideas 

□ 

1 was observed in my classroom by a coach and received feedback 

□ 

7. Which of the following did you learn during the math professional development sessions you attended? Check all activities that apply. 

1 learned how to use manipulatives (concrete objects) to teach mathematical concepts 

□ 

1 learned how to use visual representations to convey mathematical concepts (such as number line, strip 
diagrams, pictorial representations, etc.) 

□ 

1 learned howto ask students questions and provide feedback 

□ 

1 learned how to use a curriculum or how to use supplemental curricula materials 

□ 

8. Did your professional development use a commercially available mathematics professional development 
program such as DMI, NUMBERS, INTEL MATHEMATICS? 

□ Yes □ No 

If Yes, specify the name of the program 

DMI is Developing Mathematical Ideas. 

Source: Authors' creation. 


Table C2. Survey questions given to treatment teachers to determine their satisfaction with the 
Developing Mathematical Ideas program 

Survey item 

Strongly 

disagree 

Disagree 

Agree 

Strongly 

agree 

My knowledge of fractions has improved. 

1 

2 

3 

4 

My knowledge of how to teach fractions has improved. 

1 

2 

3 

4 

Attending the DMI professional development sessions was a good professional 
learning experience. 

1 

2 

3 

4 

1 was able to put into practice what 1 learned during the DMI professional development 
sessions. 

1 

2 

3 

4 

DMI is Developing Mathematical Ideas. 

Source: Authors' creation. 
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Appendix D. Data analysis 


This appendix presents data analyses procedures and findings from the main impact analy- 
ses, sensitivity analyses, and imputation analysis. In addition, the self-report data from 
teachers in the treatment and control conditions on their professional development activ- 
ities are also summarized. 

Assessing impacts at student level 

Impact analyses for student outcomes were based on a multilevel model with students at 
level 1 and schools at level 2 (table Dl). For the student impact, the student-level covari- 
ates included the previous year’s state math test score standardized within state and five 
dummy-coded covariates that identify students as in special education, talented and gifted, 
English learner, Black, or Hispanic. Five school-level covariates were also included: the 
percentage of grade 4 students who passed the state math test, the percentage of students 
who were eligible for the federal school lunch program, total number of students in grade 
4, the percentage of students who were Black, and the percentage of students who were 
Hispanic. 

Schools were assigned within school pairs and districts (that is, blocked on pairs and dis- 
tricts). The model used in the analysis — students nested within schools — is a two-level 
model that ignored school pairs and districts as blocking factors. Ignoring blocking, though 
not faithful to the actual design used, does not appear to cause problems with either the 
Type I or Type II error rate (Dong & Lipsey, 2010). The decision to use a two-level model 
is consistent with the recommendation in the literature about how to handle clustering for 
studies that randomly assign schools to conditions within districts (Raudenbush & Sadoff, 
2008; Schochet, 2008; Zhu, Jacob, Bloom, & Xu, 2011). 

The models were fitted to the data with SAS PROC MIXED version 9.2 (SAS Institute, 
2009) using restricted maximum likelihood, generally recommended for multilevel models 
(Hox, 2002). Following What Works Clearinghouse 3.0 guidelines (What Works Clearing- 
house, 2014), Hedges’ g (Hedges, 1981) was also computed for each fixed effect. 

Sensitivity analysis. A sensitivity analysis was conducted to see whether analytic models 
that include blocks produce results similar to those without blocks. A third level was added 
to the models that accounted for either school pairs or districts as either random or fixed 
blocks. The results from the models that accounted for blocks differed only slightly from 
the models that excluded blocks for students, which was expected given that the school 
pair and district were not highly correlated with outcomes (table D2). 

Assessing impacts at teacher level 

As with students, teacher-level analysis used a two-level fixed effects model, with teach- 
ers at level 1 and schools at level 2. The teacher analysis included several teacher-level 
and school-level covariates (table D3). The 11 teacher-level covariates included teacher’s 
pretest math test score, teacher’s education level (master’s vs. bachelor’s), years of expe- 
rience teaching in grade 4, years of experience teaching grades 6-12, math certification 
(yes or no), middle school certification (yes or no), special education certification (yes or 
no), number of courses in math, number of courses in math instruction, and two variables 
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Table Dl. Impact of Developing Mathematical Ideas on student fractions 
achievement, 2014/15 

Model 

Parameter 

estimates 

Standard errors 

p value 

Fixed effects 

Intercept 

18.86 

1.87*** 

<.001 

Condition 

-0.15 

0.32 

.638 

Student 

Math pretest 

3.25 

0.06*** 

<.001 

Special education status 

-0.52 

0.18** 

.005 

Talented and gifted status 

1.42 

q 20 *** 

<.001 

English learner student 

-0.58 

0.26* 

.025 

Black 

-1.06 

0.15*** 

<.001 

Hispanic 

-0.23 

0.22 

.296 

School 

Percentage of grade 4 students who passed 

state math test 

0.00 

0.02 

.862 

Percentage of students eligible for the 
federal school lunch program 

-0.04 

0 . 01 *** 

<.001 

Total number of students in grade 4 

0.00 

0.01 

.740 

Percentage of students who were Black 

0.00 

0.01 

.682 

Percentage of students who were Hispanic 

0.03 

0.01 

.089 

Variances 

School intercept 

1.84 

0.35*** 

<.001 

Residual 

11.61 

q 26*** 

<.001 

Intraclass correlation 

0.14 

na 

na 

Hedges’ g a (condition) 

-0.03 

na 

na 

p value (condition) 

.637 

na 

na 

Adjusted means 

Treatment 

16.16 

na 

na 

Control 

16.31 

na 

na 


* Significant at p < .05; ** Significant at p < .01; *** Significant at p < .001. 

DMI is Developing Mathematical Ideas; na is not applicable. 

a. Hedges’ g is an effect size index that shows the magnitude of the difference between treatment group and 
control group. 

Note: Tests of fixed effects accounted for schools as the unit of analysis within the treatment condition. Analy- 
ses conducted with a mixed-model analysis of covariance used 77 degrees of freedom. 

Source: Authors' analysis of primary data collected for the study; see appendix A. 


related to coaching. At the school level, five covariates were included: percentage of stu- 
dents who passed the state math test in grade 4, enrollment in grade 4, percentage of stu- 
dents eligible for the federal school lunch program, percentage of students who were Black, 
and percentage of students who were Hispanic. As with studentdevel analysis, the model 
used did not include district level. 

Sensitivity analysis. Sensitivity analyses were conducted at the teacher level to determine 
whether models with districts as either random effects or fixed effects produce impacts 
similar to those with no blocks. The threedevel models did not differ notably from two-lev- 
el models that excluded districts as a blocking variable (table D4). 
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Table D2. Sensitivity analyses for the impact of Developing Mathematical Ideas on 
the student posttest — Test for Understanding of Fractions, 2014/15 

Model description 

Hedges' g a 

p value 

Two-level models 

Pretest covariate 

-0.02 

.806 

Pretest and demographic covariates 

-0.01 

.936 

Pretest, school, and demographic covariates 

-0.03 

.637 

Three-level models 

Pretest, school, and demographic covariates nested within classrooms and schools 

-0.03 

.614 

Pretest, school, and demographic covariates nested within classrooms and matched 
school pairs 

-0.03 

.640 

Pretest, school, and demographic covariates nested within schools and school pairs 

-0.03 

.638 

Pretest, school, and demographic covariates with districts as random blocks 

-0.04 

.392 

Pretest, school, and demographic covariates with districts as fixed blocks 

-0.04 

.357 


Note: Analyses conducted with a mixed-model analysis of covariance used 82 degrees of freedom for the test 
of treatment condition in models 1 and 2, 77 degrees of freedom for models 3-6, 76 degrees of freedom for 
model 7, and 70 degrees of freedom for model 8. 

a. Hedges' g is an effect size index that show the magnitude of the difference between treatment group and 
control group. 

Source: Authors' analysis of primary data collected for the study; see appendix A. 


Imputation of missing student pretest data 

At the end of school year, 4,882 students from the classes of teachers in the study took the 
posttest (the Test for Understanding of Fractions). Flowever, the study team was able to 
obtain pretest scores — grade 3 spring state math achievement scores — from the districts 
for only 4,204 of these students. The analytic sample included only these 4,204 students 
with posttest and pretest scores. The study team imputed missing pretest values based on 
other available pretest data and found that it did not appreciably change the results. In 
the original analysis of covariance model with pretest as the only covariate, the difference 
between conditions was -0.38, t = -0.25, p = .806. With the imputed data, the difference 
was -0.35, t = -0.22, p = .825. 

Correlations between pretest and posttest scores 

The correlation between student state math achievement scores and student fractions 
posttest scores (on the Test for Understanding of Fractions) was .70. The correlation 
between the teacher pretest (the Mathematical Knowledge for Teaching: Number Con- 
cepts and Operations scale) and the teacher posttest (the Mathematical Knowledge for 
Teaching: Fractions scale) was correlated at .65. Both correlations were statistically signif- 
icant at p < .001. These correlations indicate that using these pre-measures in the analy- 
sis improves the precision of the statistical analyses conducted because they are suitable 
covariates. 

Contrast between the fractions professional development received by teachers in the treatment and 
control conditions 

Teachers in both the treatment and control conditions reported on the fractions profes- 
sional development they participated in during the school year from August 2014 to March 
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Table D3. Impact of Developing Mathematical Ideas on teachers’ knowledge of 
fractions, 2014/15 

Model 

Parameter 

estimates 

Standard 

errors 

p value 

Fixed effects 

Intercept 

9.73 

2.48*** 

<.001 

Condition 

0.81 

0.41 

.051 

Teacher 

Math pretest 

0.64 

0.05*** 

<.001 

Master's degree 

-0.35 

0.42 

.406 

Years of teaching grade 4 

-0.09 

0.06 

.137 

Years of teaching grade 6-12 

0.37 

0.90 

.680 

Math certification 

0.63 

0.95 

.509 

Middle school certification 

0.01 

0.65 

.993 

Special education certification 

0.20 

0.84 

.808 

Courses in math 

0.19 

0.30 

.525 

Courses in teaching math 

-0.31 

0.32 

.333 

Coaching with reduced teaching load 

0.25 

1.32 

.849 

Coaching in addition to teaching 

-1.55 

0.75* 

.040 

School 

Percentage of grade 4 students who passed state math test 

0.03 

0.02 

.189 

Percentage of students eligible for the federal school lunch program 

-0.02 

0.02 

.167 

Total number of students in grade 4 


0.01 

.977 

Percentage of students who were Black 

0.01 

0.01 

.600 

Percentage of students who were Hispanic 

0.02 

0.02 

.415 

Variances 

School intercept 

0.16 

0.63 

.795 

Residual 

9.81 

1.05*** 

<.001 

Intraclass correlation 

0.02 

na 

na 

Hedges’ g a (condition) 

0.19 

na 

na 

p value (condition) 

.051 

na 

na 

Adjusted means 

Treatment 

20.23 

na 

na 

Control 

19.42 

na 

na 


* Significant at p < .05; *** Significant at p < .001. 

DMI is Developing Mathematical Ideas; na is not applicable; .. is a negligible value. 

a. Hedges' g is an effect size index that shows the magnitude of the difference between treatment group and 
control group. 

Note: Tests of fixed effects accounted for schools as the unit of analysis within the treatment condition. Analy- 
ses conducted with a mixed-model analysis of covariance used 77 degrees of freedom. 

Source: Authors' analysis of primary data collected for the study; see appendix A. 


2015. About a third of the teachers in the control condition (45 out of 135) indicated that 
they participated in some form of fractions professional development during the school 
year. Teachers in the control condition reported spending on average 5 hours on fractions 
professional development, compared with the 20 hours reported by teachers in the treat- 
ment condition. 

While 129 teachers in the treatment condition participated in the DMI professional devel- 
opment (based on attendance data collected for the study), only 113 (88 percent) reported 
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Table D4. Sensitivity analyses for the impact of Developing Mathematical Ideas on the 
teacher posttest — Mathematical Knowledge for Teaching: Fractions scale, 2014/15 


Model description 

Hedges’ g a 

p value 

Two-level models 

Pretest covariate 

0.16 

.078 

Eleven teacher-level covariates and five school-level covariates with 

math pretest covariate 

0.19 

.051 

Three-level models with schools and districts 

Eleven teacher-level covariates and five school-level covariates with 

math pretest covariate and districts as random blocks 

0.19 

.051 

Eleven teacher-level covariates and five school-level covariates with 

math pretest covariate and districts as fixed blocks 

0.19 

.060 


Note: Analyses conducted with a mixed-model analysis of covariance used 82 degrees of freedom for the 
test of treatment in model 1, 77 degrees of freedom in model 2, 76 degrees of freedom in model 3, and 70 
degrees of freedom in model 4. 

Hedges' g is an effect size index that shows the magnitude of the difference between treatment group and 
control group. 

Source: Authors' analysis of primary data collected for the study; see appendix A. 


participating in a fractions professional development during the school year. Based on this 
underreporting by teachers in the treatment condition, it is possible that teachers in the 
control condition also underreported their participation in fractions professional develop- 
ment. Still, the self-report data from the eight monthly surveys show a contrast between 
teachers in the two conditions (table D5). 

A larger percentage of teachers in both the treatment and control conditions reported 
participating in professional development in fractions during the months of the DMI pro- 
fessional development (September to December 2014) than during the month preceding 
(August 2014) or the months following (January to March 2015). While there was no sta- 
tistically significant difference between the percentage of teachers in the two conditions 
who participated in fractions professional development during the months preceding DMI 
(j( 2 (l) = 0.2466, p = .620) or following DMI (j( 2 (l) = 0.0215, p = .883), there was a signifi- 
cant difference during the months DMI was offered (j( 2 (l) = 106.4322, p < .001). 
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Table D5. Topics and activities covered in the fractions professional development sessions attended 
by treatment and control teachers, 2014/15 



Percentage of 
teachers before DMI 
(August 2014) 

Percentage of 
teachers during DMI 
(September to 
December 2014) 

Percentage of 
teachers after DMI 
(January to 

March 2015) 

Fractions professional development 

Treatment 

condition 0 

Control 

condition 11 

Treatment 

condition 

Control 

condition 

Treatment 

condition 

Control 

condition 

Had fractions professional development 

9.91 

11.97 

87.60 

24.44 

13.95 

13.33 

Topics 

Understanding fractions as numbers 0 

6.31 

9.40 

82.95 

20.00 

10.08 

11.85 

Equivalence of fractions 0 

6.31 

11.11 

78.29 

18.52 

10.08 

12.59 

Adding and subtracting fractions 0 

5.41 

7.69 

75.19 

18.52 

10.08 

8.15 

Multiplying and dividing fractions 0 

3.60 

5.13 

77.52 

10.37 

9.30 

8.15 

Improper fractions/mixed numbers 0 

4.50 

7.69 

62.02 

16.30 

8.53 

8.15 

Activities 

Observed videos or live demonstrations of math 

teaching during the professional development session 0 

4.50 

7.69 

86.82 

16.30 

10.08 

6.67 

Worked with colleagues to plan or enhance math 
lessons/activities 0 

3.60 

7.69 

81.40 

17.04 

10.08 

11.11 

Discussed students' math work samples with 
colleagues to analyze how students think about and 

learn math 0 

0.90 

4.27 

84.50 

8.89 

10.08 

5.93 

Worked on math problems (either individually or with 
a partner) 0 

6.31 

6.84 

87.60 

19.26 

10.85 

6.67 

Engaged in group discussions about the math 
concepts being learned 0 

6.31 

5.98 

86.82 

17.78 

10.08 

8.89 

Implemented a math lesson or activity in my classroom 0 

0.90 

1.71 

83.72 

12.59 

6.20 

7.41 

Received feedback from the professional development 
staff about my understanding of mathematical ideas 0 

1.80 

2.56 

75.97 

6.67 

6.98 

2.96 

Learned how to use manipulatives (concrete objects) 
to teach math concepts 0 

5.41 

8.55 

63.57 

16.30 

9.30 

10.37 

Learned how to use visual representations to convey 
math concepts (such as number line, strip diagrams, 
pictorial representations, and so forth) 0 

4.50 

7.69 

84.50 

17.04 

8.53 

7.41 

Learned how to ask students questions and provide 

feedback 0 

2.70 

4.27 

78.29 

11.11 

6.20 

5.93 

Was required to observe demonstrations of math 
teaching in classroom settings 

0.00 

2.56 

57.36 

5.93 

3.88 

2.96 

Developed math lessons/activities outside of the 
professional development session, for use in my 
classroom 

0.90 

2.56 

55.81 

6.67 

6.20 

3.70 

Was observed in my classroom by a coach and 
received feedback 

0.00 

0.00 

8.53 

1.48 

1.55 

1.48 


DMI is Developing Mathematical Ideas. 

Note: Data are based on teacher responses to eight monthly surveys. A total of 129 treatment teachers and 135 control teachers 
responded to the monthly surveys. Control teacher response rate for the eight monthly surveys is as follows: survey 1 = 85.19 percent, 
survey 2 = 81.48 percent, survey 3 = 83.70 percent, survey 4 = 94.07 percent, survey 5 = 93.33 percent, survey 6 = 96.30 percent, 

survey 7 = 95.56 percent, survey 8 = 98.52 percent. Treatment teacher response rate for the eight monthly surveys is as follows: 

survey 1 = 89.15 percent, survey 2 = 86.05 percent, survey 3 = 86.05 percent, survey 4 = 98.45 percent, survey 5 = 99.22 percent, 

survey 6 = 97.67 percent, survey 7 = 96.90 percent, survey 8 = 96.12 percent. 

a. Based on responses of 111 treatment teachers. Teachers from one school district started DMI in August 2014 and one school dis- 
trict began participation in the study after August 2014. 

b. Based on responses of 117 control teachers. Teachers from one school district started DMI in August 2014 and one school district 
began participation in the study after August 2014. 

c. These topics and activities were addressed in the DMI professional development program. 

Source: Authors' analysis of primary data collected for the study; see appendix A. 
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Notes 


1. In 2013 all states in the REL Southeast Region used the Common Core State Stan- 
dards as their state standards. Their current state standards differ from the Common 
Core standards in some respects but continue to focus on fractions concepts in grade 4- 

2. Math educators in the region have been concerned with poor student performance in 
algebra. For example, in 2014, the pass rate in the algebra 1 end-of-course assessment 
was 68 percent in Florida (Florida Department of Education, 2016). 

3. Members of the REF Southeast Improving Mathematics Instruction Research Alli- 
ance from Georgia and South Carolina formed a work group to select a professional 
development program that stressed content knowledge of fractions and that provided 
teachers with instructional activities that could help their students develop under- 
standing of key mathematical ideas. The work group included members of the alliance 
and, in some cases, their designees (who were state education agency math leaders 
responsible for math education in their states and math specialists). The study team 
solicited information from vendors that were able to provide professional development 
to the large number of schools the staff anticipated being involved in the study — 
about 40 schools with more than 100 grade 4 math teachers in three states. Only three 
vendors indicated that they had adequate capacity to conduct this type of large-scale 
professional development, and their programs focused primarily on building content 
knowledge. 

4. The study team believes student attrition to be students missing at random. Teach- 
ers participated in the professional development effort, and students would have been 
unaware of the project. Any shifts in classroom instruction would have been subtle 
and probably undetectable for grade 4 students because textbooks and other curricular 
material would have stayed the same. 
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